2025-05-23 21:29:48,051 [ 523875 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2025-05-23 21:29:48,051 [ 523875 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2025-05-23 21:29:48,051 [ 523875 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2025-05-23 21:29:48,051 [ 523875 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_lx8764 --privileged --dns-search='.' --memory=30709035008 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel2_0.jsonl --report-log-exclude-logs-on-passed-tests test_part_log_table/test.py::test_config_without_part_log test_part_log_table/test.py::test_part_log_contains_partition test_permissions_drop_replica/test.py::test_drop_permissions test_postgresql_protocol/test.py::test_java_client test_postgresql_protocol/test.py::test_prepared_statement test_postgresql_protocol/test.py::test_psql_client test_postgresql_protocol/test.py::test_python_client test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum 'test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False]' 'test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True]' test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection 'test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False]' 'test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True]' test_quorum_inserts/test.py::test_insert_quorum_with_ttl test_quorum_inserts/test.py::test_simple_add_replica test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers test_reload_clusters_config/test.py::test_add_cluster test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_simple_reload test_reload_clusters_config/test.py::test_update_one_cluster 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys test_replicated_merge_tree_thread_schedule_timeouts/test.py::test_query_exception_on_thread_pool_full test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_auxiliary_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_default_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_not_exists_auxiliary_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_drop_replicated_merge_tree_with_auxiliary_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_path_ambiguity test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_hosts_limit test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_distributed_s3_table_engine 'test_s3_cluster/test.py::test_hive_partitioning[0]' 'test_s3_cluster/test.py::test_hive_partitioning[1]' test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster test_session_log/test.py::test_grpc_session test_session_log/test.py::test_mysql_session test_session_log/test.py::test_parallel_sessions test_session_log/test.py::test_postgres_session test_settings_from_server/test.py::test_settings_from_server test_settings_randomization/test.py::test_settings_randomization test_shard_level_const_function/test.py::test_remote test_sharding_key_from_default_column/test.py::test_default_column test_sharding_key_from_default_column/test.py::test_materialized_column_allow_insert_materialized test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized_different_shards test_ssl_cert_authentication/test.py::test_create_user test_ssl_cert_authentication/test.py::test_https test_ssl_cert_authentication/test.py::test_https_non_ssl_auth test_ssl_cert_authentication/test.py::test_https_wrong_cert test_ssl_cert_authentication/test.py::test_native test_ssl_cert_authentication/test.py::test_native_fallback_to_password test_ssl_cert_authentication/test.py::test_native_wrong_cert test_ssl_cert_authentication/test.py::test_x509_san_support test_ssl_cert_authentication/test.py::test_x509_san_wildcard_support 'test_storage_mongodb_legacy/test.py::test_arrays[False]' 'test_storage_mongodb_legacy/test.py::test_auth_source[False]' 'test_storage_mongodb_legacy/test.py::test_complex_data_type[False]' 'test_storage_mongodb_legacy/test.py::test_incorrect_data_type[False]' 'test_storage_mongodb_legacy/test.py::test_missing_columns[False]' 'test_storage_mongodb_legacy/test.py::test_no_credentials[False]' 'test_storage_mongodb_legacy/test.py::test_predefined_connection_configuration[False]' 'test_storage_mongodb_legacy/test.py::test_secure_connection[True]' -vvv " altinityinfra/integration-tests-runner:ad96270260ff '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_rocksdb_options/test.py::test_invalid_block_based_table_options test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_auxiliary_zookeeper test_postgresql_protocol/test.py::test_java_client test_s3_cluster/test.py::test_ambiguous_join test_storage_mongodb_legacy/test.py::test_arrays[False] test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum test_replication_credentials/test.py::test_credentials_and_no_credentials test_ssl_cert_authentication/test.py::test_create_user test_reload_clusters_config/test.py::test_add_cluster [gw2] [ 1%] PASSED test_ssl_cert_authentication/test.py::test_create_user test_ssl_cert_authentication/test.py::test_https [gw2] [ 2%] PASSED test_ssl_cert_authentication/test.py::test_https test_ssl_cert_authentication/test.py::test_https_non_ssl_auth [gw2] [ 3%] PASSED test_ssl_cert_authentication/test.py::test_https_non_ssl_auth test_ssl_cert_authentication/test.py::test_https_wrong_cert [gw2] [ 4%] PASSED test_ssl_cert_authentication/test.py::test_https_wrong_cert test_ssl_cert_authentication/test.py::test_native [gw2] [ 5%] PASSED test_ssl_cert_authentication/test.py::test_native test_ssl_cert_authentication/test.py::test_native_fallback_to_password [gw2] [ 6%] PASSED test_ssl_cert_authentication/test.py::test_native_fallback_to_password test_ssl_cert_authentication/test.py::test_native_wrong_cert [gw2] [ 7%] PASSED test_ssl_cert_authentication/test.py::test_native_wrong_cert test_ssl_cert_authentication/test.py::test_x509_san_support [gw2] [ 8%] PASSED test_ssl_cert_authentication/test.py::test_x509_san_support test_ssl_cert_authentication/test.py::test_x509_san_wildcard_support [gw2] [ 9%] PASSED test_ssl_cert_authentication/test.py::test_x509_san_wildcard_support test_session_log/test.py::test_grpc_session [gw1] [ 10%] PASSED test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options [gw8] [ 11%] PASSED test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials [gw6] [ 12%] PASSED test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] [gw0] [ 13%] PASSED test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression [gw6] [ 14%] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] [gw0] [ 15%] FAILED test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection [gw5] [ 16%] PASSED test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_auxiliary_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_default_zookeeper [gw0] [ 17%] FAILED test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_hosts_limit [gw6] [ 18%] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection [gw0] [ 19%] FAILED test_s3_cluster/test.py::test_cluster_hosts_limit test_s3_cluster/test.py::test_cluster_with_header [gw8] [ 20%] PASSED test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials [gw0] [ 21%] FAILED test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection [gw0] [ 22%] FAILED test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count [gw0] [ 23%] FAILED test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro [gw0] [ 24%] FAILED test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated [gw1] [ 25%] PASSED test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options [gw0] [ 26%] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_distributed_s3_table_engine [gw8] [ 27%] PASSED test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials [gw0] [ 28%] FAILED test_s3_cluster/test.py::test_distributed_s3_table_engine test_s3_cluster/test.py::test_hive_partitioning[0] [gw3] [ 29%] PASSED test_storage_mongodb_legacy/test.py::test_arrays[False] test_storage_mongodb_legacy/test.py::test_auth_source[False] [gw3] [ 30%] PASSED test_storage_mongodb_legacy/test.py::test_auth_source[False] test_storage_mongodb_legacy/test.py::test_complex_data_type[False] [gw5] [ 31%] PASSED test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_default_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_not_exists_auxiliary_zookeeper [gw5] [ 32%] PASSED test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_not_exists_auxiliary_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_drop_replicated_merge_tree_with_auxiliary_zookeeper [gw3] [ 33%] PASSED test_storage_mongodb_legacy/test.py::test_complex_data_type[False] test_storage_mongodb_legacy/test.py::test_incorrect_data_type[False] [gw0] [ 34%] FAILED test_s3_cluster/test.py::test_hive_partitioning[0] test_s3_cluster/test.py::test_hive_partitioning[1] [gw3] [ 35%] PASSED test_storage_mongodb_legacy/test.py::test_incorrect_data_type[False] test_storage_mongodb_legacy/test.py::test_missing_columns[False] [gw8] [ 36%] PASSED test_replication_credentials/test.py::test_same_credentials [gw3] [ 37%] PASSED test_storage_mongodb_legacy/test.py::test_missing_columns[False] test_storage_mongodb_legacy/test.py::test_no_credentials[False] [gw2] [ 38%] PASSED test_session_log/test.py::test_grpc_session test_session_log/test.py::test_mysql_session [gw3] [ 39%] PASSED test_storage_mongodb_legacy/test.py::test_no_credentials[False] test_storage_mongodb_legacy/test.py::test_predefined_connection_configuration[False] [gw3] [ 40%] PASSED test_storage_mongodb_legacy/test.py::test_predefined_connection_configuration[False] test_storage_mongodb_legacy/test.py::test_secure_connection[True] [gw2] [ 41%] PASSED test_session_log/test.py::test_mysql_session test_session_log/test.py::test_parallel_sessions [gw4] [ 42%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] [gw4] [ 43%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format [gw0] [ 44%] FAILED test_s3_cluster/test.py::test_hive_partitioning[1] test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference [gw2] [ 45%] PASSED test_session_log/test.py::test_parallel_sessions test_session_log/test.py::test_postgres_session [gw0] [ 46%] PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference test_s3_cluster/test.py::test_remote_hedged [gw0] [ 47%] FAILED test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged [gw4] [ 48%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] [gw4] [ 49%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] [gw0] [ 50%] FAILED test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all [gw2] [ 51%] PASSED test_session_log/test.py::test_postgres_session [gw0] [ 52%] FAILED test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards [gw0] [ 53%] FAILED test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all [gw4] [ 54%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] [gw0] [ 55%] FAILED test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards [gw0] [ 56%] FAILED test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster [gw0] [ 57%] FAILED test_s3_cluster/test.py::test_wrong_cluster [gw1] [ 58%] PASSED test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options [gw5] [ 59%] PASSED test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_drop_replicated_merge_tree_with_auxiliary_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_path_ambiguity [gw4] [ 60%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] [gw5] [ 61%] PASSED test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_path_ambiguity [gw4] [ 62%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] test_permissions_drop_replica/test.py::test_drop_permissions [gw4] [ 63%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format [gw4] [ 64%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_sharding_key_from_default_column/test.py::test_default_column test_part_log_table/test.py::test_config_without_part_log [gw6] [ 65%] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] [gw6] [ 66%] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] [gw1] [ 67%] PASSED test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options [gw6] [ 68%] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] test_quorum_inserts/test.py::test_insert_quorum_with_ttl [gw5] [ 69%] PASSED test_part_log_table/test.py::test_config_without_part_log test_part_log_table/test.py::test_part_log_contains_partition [gw5] [ 70%] PASSED test_part_log_table/test.py::test_part_log_contains_partition [gw1] [ 71%] PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options [gw8] [ 72%] PASSED test_sharding_key_from_default_column/test.py::test_default_column test_sharding_key_from_default_column/test.py::test_materialized_column_allow_insert_materialized test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility [gw8] [ 73%] PASSED test_sharding_key_from_default_column/test.py::test_materialized_column_allow_insert_materialized test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized [gw2] [ 74%] PASSED test_permissions_drop_replica/test.py::test_drop_permissions [gw8] [ 75%] PASSED test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized_different_shards [gw3] [ 76%] PASSED test_storage_mongodb_legacy/test.py::test_secure_connection[True] [gw8] [ 77%] PASSED test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized_different_shards [gw6] [ 78%] PASSED test_quorum_inserts/test.py::test_insert_quorum_with_ttl test_quorum_inserts/test.py::test_simple_add_replica test_settings_from_server/test.py::test_settings_from_server [gw1] [ 79%] PASSED test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options [gw1] [ 80%] PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options [gw1] [ 81%] PASSED test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options [gw1] [ 82%] PASSED test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options [gw6] [ 83%] PASSED test_quorum_inserts/test.py::test_simple_add_replica [gw1] [ 84%] PASSED test_rocksdb_options/test.py::test_valid_options test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys [gw4] [ 85%] PASSED test_settings_from_server/test.py::test_settings_from_server test_shard_level_const_function/test.py::test_remote test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers [gw3] [ 86%] PASSED test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys [gw3] [ 87%] PASSED test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys [gw0] [ 88%] PASSED test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility [gw4] [ 89%] PASSED test_shard_level_const_function/test.py::test_remote test_replicated_merge_tree_thread_schedule_timeouts/test.py::test_query_exception_on_thread_pool_full [gw2] [ 90%] PASSED test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers test_settings_randomization/test.py::test_settings_randomization [gw2] [ 91%] PASSED test_settings_randomization/test.py::test_settings_randomization [gw0] [ 92%] PASSED test_replicated_merge_tree_thread_schedule_timeouts/test.py::test_query_exception_on_thread_pool_full [gw7] [ 93%] PASSED test_postgresql_protocol/test.py::test_java_client test_postgresql_protocol/test.py::test_prepared_statement [gw7] [ 94%] PASSED test_postgresql_protocol/test.py::test_prepared_statement test_postgresql_protocol/test.py::test_psql_client [gw7] [ 95%] PASSED test_postgresql_protocol/test.py::test_psql_client test_postgresql_protocol/test.py::test_python_client [gw7] [ 96%] PASSED test_postgresql_protocol/test.py::test_python_client [gw9] [ 97%] PASSED test_reload_clusters_config/test.py::test_add_cluster test_reload_clusters_config/test.py::test_delete_cluster [gw9] [ 98%] FAILED test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_simple_reload [gw9] [ 99%] PASSED test_reload_clusters_config/test.py::test_simple_reload test_reload_clusters_config/test.py::test_update_one_cluster [gw9] [100%] FAILED test_reload_clusters_config/test.py::test_update_one_cluster =================================== FAILURES =================================== _______________________ test_cluster_default_expression ________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_cluster_default_expression(started_cluster): node = started_cluster.instances["s0_0_0"] node.query( f"insert into function s3('http://minio1:9001/root/data/data1', 'minio', '{minio_secret_key}', JSONEachRow) select 1 as id settings s3_truncate_on_insert=1" ) node.query( f"insert into function s3('http://minio1:9001/root/data/data2', 'minio', '{minio_secret_key}', JSONEachRow) select * from numbers(0) settings s3_truncate_on_insert=1" ) node.query( f"insert into function s3('http://minio1:9001/root/data/data3', 'minio', '{minio_secret_key}', JSONEachRow) select 2 as id settings s3_truncate_on_insert=1" ) expected_result = node.query( f"SELECT * FROM s3('http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id" ) result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id" ) assert result == expected_result result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'auto', 'id UInt32, date Date DEFAULT 18262') order by id" ) assert result == expected_result result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id" ) assert result == expected_result result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'auto', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id" ) assert result == expected_result result = node.query( "SELECT * FROM s3Cluster(cluster_simple, test_s3_with_default) order by id" ) assert result == expected_result > result = node.query( f"""SELECT * FROM s3('http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id SETTINGS object_storage_cluster = 'cluster_simple'""" ) test_s3_cluster/test.py:635: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007fe48149eac3 E 16. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query insert into function s3('http://minio1:9001/root/data/data1', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select 1 as id settings s3_truncate_on_insert=1 on s0_0_0 Executing query insert into function s3('http://minio1:9001/root/data/data2', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select * from numbers(0) settings s3_truncate_on_insert=1 on s0_0_0 Executing query insert into function s3('http://minio1:9001/root/data/data3', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select 2 as id settings s3_truncate_on_insert=1 on s0_0_0 Executing query SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'auto', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'auto', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, test_s3_with_default) order by id on s0_0_0 Executing query SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:18 [ 684 ] DEBUG : Executing query insert into function s3('http://minio1:9001/root/data/data1', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select 1 as id settings s3_truncate_on_insert=1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:18 [ 684 ] DEBUG : Executing query insert into function s3('http://minio1:9001/root/data/data2', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select * from numbers(0) settings s3_truncate_on_insert=1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:18 [ 684 ] DEBUG : Executing query insert into function s3('http://minio1:9001/root/data/data3', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select 2 as id settings s3_truncate_on_insert=1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:18 [ 684 ] DEBUG : Executing query SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:19 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:19 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'auto', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:19 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:19 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'auto', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:19 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, test_s3_with_default) order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:19 [ 684 ] DEBUG : Executing query SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) ________________________ test_cluster_format_detection _________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_cluster_format_detection(started_cluster): node = started_cluster.instances["s0_0_0"] expected_desc_result = node.query( f"desc s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}', 'CSV')" ) desc_result = node.query( f"desc s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}')" ) assert expected_desc_result == desc_result expected_result = node.query( f"SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}', 'CSV', 'a String, b UInt64') order by a, b" ) result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2" ) assert result == expected_result result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}', auto, 'a String, b UInt64') order by a, b" ) assert result == expected_result > result = node.query( """SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2 SETTINGS object_storage_cluster = 'cluster_simple'""" ) test_s3_cluster/test.py:573: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007fe48149eac3 E 16. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2 E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query desc s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') on s0_0_0 Executing query desc s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd') on s0_0_0 Executing query SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'a String, b UInt64') order by a, b on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd') order by c1, c2 on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', auto, 'a String, b UInt64') order by a, b on s0_0_0 Executing query SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2 SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:19 [ 684 ] DEBUG : Executing query desc s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:19 [ 684 ] DEBUG : Executing query desc s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:19 [ 684 ] DEBUG : Executing query SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'a String, b UInt64') order by a, b on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:20 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd') order by c1, c2 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:20 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', auto, 'a String, b UInt64') order by a, b on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:20 [ 684 ] DEBUG : Executing query SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2 SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) ___________________________ test_cluster_hosts_limit ___________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_cluster_hosts_limit(started_cluster): node = started_cluster.instances["s0_0_0"] query_id_def = str(uuid.uuid4()) resp_def = node.query( f""" SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) """, query_id=query_id_def, ) # object_storage_max_nodes is greater than number of hosts in cluster query_id_4_hosts = str(uuid.uuid4()) > resp_4_hosts = node.query( f""" SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_max_nodes=4 """, query_id=query_id_4_hosts, ) test_s3_cluster/test.py:1007: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_max_nodes is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007fe48149eac3 E 16. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: SELECT * from s3Cluster( E 'cluster_simple', E 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) E SETTINGS object_storage_max_nodes=4 E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_max_nodes=4 on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:20 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:20 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_max_nodes=4 on s0_0_0 (cluster.py:3570, query) ___________________________ test_cluster_with_header ___________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_cluster_with_header(started_cluster): node = started_cluster.instances["s0_0_0"] assert ( node.query( "SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'))" ) == "SomeValue\n" ) assert ( node.query( "SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV')" ) == "SomeValue\n" ) assert ( node.query( "SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'))" ) == "SomeValue\n" ) assert ( node.query( "SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV')" ) == "SomeValue\n" ) > assert ( node.query( """SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) SETTINGS object_storage_cluster = 'cluster_simple'""" ) == "SomeValue\n" ) test_s3_cluster/test.py:496: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007fe48149eac3 E 16. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) on s0_0_0 Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV') on s0_0_0 Executing query SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) on s0_0_0 Executing query SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV') on s0_0_0 Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:20 [ 684 ] DEBUG : Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:20 [ 684 ] DEBUG : Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:20 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:21 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:21 [ 684 ] DEBUG : Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) ______________________ test_cluster_with_named_collection ______________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_cluster_with_named_collection(started_cluster): node = started_cluster.instances["s0_0_0"] pure_s3 = node.query("""SELECT * from s3(test_s3) ORDER BY (c1, c2, c3)""") s3_cluster = node.query( """SELECT * from s3Cluster(cluster_simple, test_s3) ORDER BY (c1, c2, c3)""" ) assert TSV(pure_s3) == TSV(s3_cluster) s3_cluster = node.query( """SELECT * from s3Cluster(cluster_simple, test_s3, structure='auto') ORDER BY (c1, c2, c3)""" ) assert TSV(pure_s3) == TSV(s3_cluster) > s3_cluster = node.query( """SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) SETTINGS object_storage_cluster = 'cluster_simple'""" ) test_s3_cluster/test.py:529: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007fe48149eac3 E 16. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) on s0_0_0 Executing query SELECT * from s3Cluster(cluster_simple, test_s3) ORDER BY (c1, c2, c3) on s0_0_0 Executing query SELECT * from s3Cluster(cluster_simple, test_s3, structure='auto') ORDER BY (c1, c2, c3) on s0_0_0 Executing query SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:21 [ 684 ] DEBUG : Executing query SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:21 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster(cluster_simple, test_s3) ORDER BY (c1, c2, c3) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:21 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster(cluster_simple, test_s3, structure='auto') ORDER BY (c1, c2, c3) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:21 [ 684 ] DEBUG : Executing query SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) __________________________________ test_count __________________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_count(started_cluster): node = started_cluster.instances["s0_0_0"] pure_s3 = node.query( f""" SELECT count(*) from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')""" ) # print(pure_s3) s3_distributed = node.query( f""" SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')""" ) # print(s3_distributed) s3_distributed_alt_syntax = node.query( f""" SELECT count(*) from s3( > 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_simple'""" ) E NameError: name 'clickhouse' is not defined test_s3_cluster/test.py:161: NameError ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Executing query SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:21 [ 684 ] DEBUG : Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:21 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3570, query) _______________________________ test_count_macro _______________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_count_macro(started_cluster): node = started_cluster.instances["s0_0_0"] s3_macro = node.query( f""" SELECT count(*) from s3Cluster( '{{default_cluster_macro}}', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')""" ) # print(s3_distributed) s3_distributed = node.query( f""" SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')""" ) # print(s3_distributed) > s3_distributed_alt_syntax = node.query( f""" SELECT count(*) from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_simple'""" ) test_s3_cluster/test.py:190: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007fe48149eac3 E 16. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: SELECT count(*) from s3( E 'http://minio1:9001/root/data/{clickhouse,database}/*', E 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3Cluster( '{default_cluster_macro}', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Executing query SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:22 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( '{default_cluster_macro}', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:22 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:22 [ 684 ] DEBUG : Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) _______________________ test_distributed_s3_table_engine _______________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_distributed_s3_table_engine(started_cluster): node = started_cluster.instances["s0_0_0"] resp_def = node.query( f""" SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) """ ) node.query("DROP TABLE IF EXISTS single_node"); node.query( f""" CREATE TABLE single_node (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV') """ ) query_id_engine_single_node = str(uuid.uuid4()) resp_engine_single_node = node.query( """ SELECT * FROM single_node ORDER BY (name, value, polygon) """, query_id = query_id_engine_single_node ) assert resp_def == resp_engine_single_node node.query("DROP TABLE IF EXISTS distributed"); > node.query( f""" CREATE TABLE distributed (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV') SETTINGS object_storage_cluster='cluster_simple' """ ) test_s3_cluster/test.py:954: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Unknown setting 'object_storage_cluster'. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x0000000009f80b8b E 3. DB::BaseSettingsHelpers::throwSettingNotFound(std::basic_string_view>) @ 0x000000001259bd63 E 4. DB::BaseSettings::set(std::basic_string_view>, DB::Field const&) @ 0x0000000011e0b6d4 E 5. DB::(anonymous namespace)::createStorageObjectStorage(DB::StorageFactory::Arguments const&, std::shared_ptr) @ 0x000000001430239e E 6. std::shared_ptr std::__function::__policy_invoker (DB::StorageFactory::Arguments const&)>::__call_impl[abi:ne190107] (DB::StorageFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::StorageFactory::Arguments const&) @ 0x0000000014302f01 E 7. DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr, std::shared_ptr, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x0000000013f81b9f E 8. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr>&, DB::LoadingStrictnessLevel) @ 0x000000001353c557 E 9. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x00000000135311c2 E 10. DB::InterpreterCreateQuery::execute() @ 0x00000000135446d8 E 11. DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000001396de0b E 12. DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000013968824 E 13. DB::TCPHandler::runImpl() @ 0x0000000014c9d90c E 14. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 15. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 16. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 17. Poco::PooledThread::run() @ 0x00000000183b975b E 18. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 19. ? @ 0x00007fe48149eac3 E 20. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: CREATE TABLE distributed E (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) E ENGINE=S3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') E SETTINGS object_storage_cluster='cluster_simple' E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 Executing query DROP TABLE IF EXISTS single_node on s0_0_0 Executing query CREATE TABLE single_node (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') on s0_0_0 Executing query SELECT * FROM single_node ORDER BY (name, value, polygon) on s0_0_0 Executing query DROP TABLE IF EXISTS distributed on s0_0_0 Executing query CREATE TABLE distributed (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') SETTINGS object_storage_cluster='cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:23 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:23 [ 684 ] DEBUG : Executing query DROP TABLE IF EXISTS single_node on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:24 [ 684 ] DEBUG : Executing query CREATE TABLE single_node (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:24 [ 684 ] DEBUG : Executing query SELECT * FROM single_node ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:24 [ 684 ] DEBUG : Executing query DROP TABLE IF EXISTS distributed on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:24 [ 684 ] DEBUG : Executing query CREATE TABLE distributed (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') SETTINGS object_storage_cluster='cluster_simple' on s0_0_0 (cluster.py:3570, query) __________________________ test_hive_partitioning[0] ___________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = allow_experimental_analyzer = 0 @pytest.mark.parametrize("allow_experimental_analyzer", [0, 1]) def test_hive_partitioning(started_cluster, allow_experimental_analyzer): node = started_cluster.instances["s0_0_0"] node.query(f"SET allow_experimental_analyzer = {allow_experimental_analyzer}") for i in range(1, 5): exists = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key={i}/*', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV """ ) if int(exists) == 0: node.query( f""" INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key={i}/data.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') SELECT {i}, {i} SETTINGS use_hive_partitioning = 0 """ ) query_id_full = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 """, query_id=query_id_full, ) result = int(result) assert result == 2 query_id_optimized = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 """, query_id=query_id_optimized, ) result = int(result) assert result == 2 query_id_cluster_full = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 """, query_id=query_id_cluster_full, ) result = int(result) assert result == 2 query_id_cluster_optimized = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 """, query_id=query_id_cluster_optimized, ) result = int(result) assert result == 2 node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple'") full_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_full}' FORMAT TSV """ ) full_traffic = int(full_traffic) assert full_traffic > 0 # 612*4 optimized_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_optimized}' FORMAT TSV """ ) optimized_traffic = int(optimized_traffic) assert optimized_traffic > 0 # 612*2 assert full_traffic > optimized_traffic cluster_full_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_cluster_full}' FORMAT TSV """ ) cluster_full_traffic = int(cluster_full_traffic) assert cluster_full_traffic == full_traffic cluster_optimized_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_cluster_optimized}' FORMAT TSV """ ) cluster_optimized_traffic = int(cluster_optimized_traffic) > assert cluster_optimized_traffic == optimized_traffic E assert 2984 == 1492 test_s3_cluster/test.py:919: AssertionError ----------------------------- Captured stderr call ----------------------------- Executing query SET allow_experimental_analyzer = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=1/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 1, 1 SETTINGS use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=2/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=2/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 2, 2 SETTINGS use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=3/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=3/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 3, 3 SETTINGS use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=4/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=4/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 4, 4 SETTINGS use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 Executing query SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple' on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='21b4b4d6-19b6-45e5-9598-315f7ed5f351' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='166cadf2-320a-4022-a485-2fd879ee5bcb' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='5f5e957f-3d5e-4aed-90bd-8c2afd9c01e8' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='8ca844d2-c805-4928-a160-5b9aa5eb7680' FORMAT TSV on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:24 [ 684 ] DEBUG : Executing query SET allow_experimental_analyzer = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:24 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:24 [ 684 ] DEBUG : Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=1/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 1, 1 SETTINGS use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:24 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=2/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:24 [ 684 ] DEBUG : Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=2/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 2, 2 SETTINGS use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:25 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=3/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:25 [ 684 ] DEBUG : Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=3/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 3, 3 SETTINGS use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:25 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=4/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:25 [ 684 ] DEBUG : Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=4/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 4, 4 SETTINGS use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:25 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:25 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:25 [ 684 ] DEBUG : Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:25 [ 684 ] DEBUG : Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:25 [ 684 ] DEBUG : Executing query SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple' on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:26 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='21b4b4d6-19b6-45e5-9598-315f7ed5f351' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:26 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='166cadf2-320a-4022-a485-2fd879ee5bcb' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:26 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='5f5e957f-3d5e-4aed-90bd-8c2afd9c01e8' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:26 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='8ca844d2-c805-4928-a160-5b9aa5eb7680' FORMAT TSV on s0_0_0 (cluster.py:3570, query) __________________________ test_hive_partitioning[1] ___________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = allow_experimental_analyzer = 1 @pytest.mark.parametrize("allow_experimental_analyzer", [0, 1]) def test_hive_partitioning(started_cluster, allow_experimental_analyzer): node = started_cluster.instances["s0_0_0"] node.query(f"SET allow_experimental_analyzer = {allow_experimental_analyzer}") for i in range(1, 5): exists = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key={i}/*', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV """ ) if int(exists) == 0: node.query( f""" INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key={i}/data.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') SELECT {i}, {i} SETTINGS use_hive_partitioning = 0 """ ) query_id_full = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 """, query_id=query_id_full, ) result = int(result) assert result == 2 query_id_optimized = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 """, query_id=query_id_optimized, ) result = int(result) assert result == 2 query_id_cluster_full = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 """, query_id=query_id_cluster_full, ) result = int(result) assert result == 2 query_id_cluster_optimized = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 """, query_id=query_id_cluster_optimized, ) result = int(result) assert result == 2 node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple'") full_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_full}' FORMAT TSV """ ) full_traffic = int(full_traffic) assert full_traffic > 0 # 612*4 optimized_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_optimized}' FORMAT TSV """ ) optimized_traffic = int(optimized_traffic) assert optimized_traffic > 0 # 612*2 assert full_traffic > optimized_traffic cluster_full_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_cluster_full}' FORMAT TSV """ ) cluster_full_traffic = int(cluster_full_traffic) assert cluster_full_traffic == full_traffic cluster_optimized_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_cluster_optimized}' FORMAT TSV """ ) cluster_optimized_traffic = int(cluster_optimized_traffic) > assert cluster_optimized_traffic == optimized_traffic E assert 2984 == 1492 test_s3_cluster/test.py:919: AssertionError ----------------------------- Captured stderr call ----------------------------- Executing query SET allow_experimental_analyzer = 1 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=2/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=3/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=4/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 Executing query SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple' on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='23809825-5fe1-46ea-a9e1-23c202c068db' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='f4a281b1-586f-433d-b2ef-9c5d54c33a10' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='4c36455a-3f9e-4ab4-a06e-23bd57cefdf1' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='f695bbbb-2593-42db-8859-106dc409f008' FORMAT TSV on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:26 [ 684 ] DEBUG : Executing query SET allow_experimental_analyzer = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:26 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:26 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=2/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:26 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=3/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:27 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=4/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:27 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:27 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:27 [ 684 ] DEBUG : Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:27 [ 684 ] DEBUG : Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:27 [ 684 ] DEBUG : Executing query SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple' on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:27 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='23809825-5fe1-46ea-a9e1-23c202c068db' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:27 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='f4a281b1-586f-433d-b2ef-9c5d54c33a10' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:28 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='4c36455a-3f9e-4ab4-a06e-23bd57cefdf1' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:28 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='f695bbbb-2593-42db-8859-106dc409f008' FORMAT TSV on s0_0_0 (cluster.py:3570, query) ______________________________ test_remote_hedged ______________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_remote_hedged(started_cluster): node = started_cluster.instances["s0_0_0"] > pure_s3 = node.query( f""" SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', 'minio{minio_secret_key}123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 """ ) test_s3_cluster/test.py:673: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 243, stderr: Received exception from server (version 25.3.3): E Code: 499. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Could not list objects in bucket 'root' with prefix 'data/', S3 exception: `SignatureDoesNotMatch`, message: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.'. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::S3Exception::S3Exception(Aws::S3::S3Errors, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&, String&&) @ 0x0000000011fea552 E 3. DB::(anonymous namespace)::S3IteratorAsync::getBatchAndCheckNext(std::vector, std::allocator>>&) @ 0x0000000011fea372 E 4. DB::IObjectStorageIteratorAsync::BatchAndHasNext std::__function::__policy_invoker::__call_impl[abi:ne190107]>(std::__function::__policy_storage const*) @ 0x000000001202c061 E 5. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, DB::IObjectStorageIteratorAsync::BatchAndHasNext ()>::operator()() @ 0x000000001202b86d E 6. void std::__function::__policy_invoker::__call_impl[abi:ne190107] (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda0'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001202bb40 E 7. ThreadPoolImpl>::ThreadFromThreadPool::worker() @ 0x000000000f5eba9b E 8. void std::__function::__policy_invoker::__call_impl[abi:ne190107]::ThreadFromGlobalPoolImpl>::ThreadFromThreadPool::*)(), ThreadPoolImpl>::ThreadFromThreadPool*>(void (ThreadPoolImpl>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000f5f25c2 E 9. ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x000000000f5e8dcf E 10. void* std::__thread_proxy[abi:ne190107]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000000f5f009a E 11. ? @ 0x00007fe48149eac3 E 12. ? @ 0x00007fe481530850 E E Job's origin stack trace: E 0. void boost::heap::priority_queue, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function&&, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000000f5f0cd7 E 1. void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool) @ 0x000000000f5ece78 E 2. std::future std::__function::__policy_invoker (std::function&&, Priority)>::__call_impl[abi:ne190107] (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority), std::future (std::function&&, Priority)>>(std::__function::__policy_storage const*, std::function&&, Priority&&) @ 0x000000001202af05 E 3. DB::IObjectStorageIteratorAsync::nextBatch() @ 0x0000000012029df0 E 4. DB::IObjectStorageIteratorAsync::isValid() @ 0x000000001202a5e2 E 5. DB::StorageObjectStorageSource::GlobIterator::estimatedKeysCount() @ 0x0000000011e6268a E 6. DB::(anonymous namespace)::ReadFromObjectStorageStep::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) (.764af0d3dc5e7addc6ed9b7904a05578) @ 0x0000000011e04bd9 E 7. DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x00000000151822ad E 8. DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x00000000151b3e4e E 9. DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000135c4e71 E 10. DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000135c479e E 11. DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000001396de0b E 12. DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000013968824 E 13. DB::TCPHandler::runImpl() @ 0x0000000014c9d90c E 14. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 15. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 16. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 17. Poco::PooledThread::run() @ 0x00000000183b975b E 18. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 19. ? @ 0x00007fe48149eac3 E 20. ? @ 0x00007fe481530850 E . (S3_ERROR) E (query: SELECT * from s3( E 'http://minio1:9001/root/data/{clickhouse,database}/*', E 'minio', 'minioClickHouse_Minio_P@ssw0rd123', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E ORDER BY (name, value, polygon) E LIMIT 1 E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minioClickHouse_Minio_P@ssw0rd123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:29 [ 684 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minioClickHouse_Minio_P@ssw0rd123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 (cluster.py:3570, query) ____________________________ test_remote_no_hedged _____________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_remote_no_hedged(started_cluster): node = started_cluster.instances["s0_0_0"] pure_s3 = node.query( f""" SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 """ ) > s3_distributed = node.query( f""" SELECT * from remote('s0_0_1', s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) ORDER BY (name, value, polygon) LIMIT 1 SETTINGS use_hedged_requests=False """ ) test_s3_cluster/test.py:710: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 49, stderr: Received exception from server (version 25.3.3): E Code: 49. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Distributed task iterator is not initialized: While executing Remote. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009f8d62b E 3. DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x0000000012840784 E 4. DB::RemoteQueryExecutor::readAsync() @ 0x0000000012841cb3 E 5. DB::RemoteSource::tryGenerate() @ 0x00000000150b06b0 E 6. DB::ISource::work() @ 0x0000000014d3cc87 E 7. DB::ExecutionThreadContext::executeTask() @ 0x0000000014d58876 E 8. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x0000000014d4c364 E 9. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000014d4b53e E 10. void std::__function::__policy_invoker::__call_impl[abi:ne190107]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000014d5c011 E 11. ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x000000000f5e8dcf E 12. void* std::__thread_proxy[abi:ne190107]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000000f5f009a E 13. ? @ 0x00007fe48149eac3 E 14. ? @ 0x00007fe481530850 E E Job's origin stack trace: E 0. void boost::heap::priority_queue, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function&&, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000000f5f0cd7 E 1. void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool) @ 0x000000000f5ea0b8 E 2. DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x0000000014d5ad53 E 3. DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x0000000014d5b299 E 4. DB::TCPHandler::runImpl() @ 0x0000000014c9dd34 E 5. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 6. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 7. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 8. Poco::PooledThread::run() @ 0x00000000183b975b E 9. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 10. ? @ 0x00007fe48149eac3 E 11. ? @ 0x00007fe481530850 E . (LOGICAL_ERROR) E (query: SELECT * from remote('s0_0_1', s3Cluster( E 'cluster_simple', E 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) E ORDER BY (name, value, polygon) E LIMIT 1 E SETTINGS use_hedged_requests=False E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 Executing query SELECT * from remote('s0_0_1', s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) ORDER BY (name, value, polygon) LIMIT 1 SETTINGS use_hedged_requests=False on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:29 [ 684 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:29 [ 684 ] DEBUG : Executing query SELECT * from remote('s0_0_1', s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) ORDER BY (name, value, polygon) LIMIT 1 SETTINGS use_hedged_requests=False on s0_0_0 (cluster.py:3570, query) _______________________________ test_select_all ________________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_select_all(started_cluster): node = started_cluster.instances["s0_0_0"] pure_s3 = node.query( f""" SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon)""" ) # print(pure_s3) s3_distributed = node.query( f""" SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon)""" ) # print(s3_distributed) > s3_distributed_alt_syntax = node.query( f""" SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple'""" ) test_s3_cluster/test.py:128: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007fe48149eac3 E 16. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: SELECT * from s3( E 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:29 [ 684 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:30 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:30 [ 684 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) _________________________ test_skip_unavailable_shards _________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_skip_unavailable_shards(started_cluster): node = started_cluster.instances["s0_0_0"] result = node.query( f""" SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1 """ ) assert result == "10\n" > result = node.query( f""" SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1, object_storage_cluster = 'cluster_non_existent_port' """ ) test_s3_cluster/test.py:328: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007fe48149eac3 E 16. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: SELECT count(*) from s3( E 'http://minio1:9001/root/data/clickhouse/part1.csv', E 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E SETTINGS skip_unavailable_shards = 1, object_storage_cluster = 'cluster_non_existent_port' E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1 on s0_0_0 Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1, object_storage_cluster = 'cluster_non_existent_port' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:30 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:30 [ 684 ] DEBUG : Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1, object_storage_cluster = 'cluster_non_existent_port' on s0_0_0 (cluster.py:3570, query) ________________________________ test_union_all ________________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_union_all(started_cluster): node = started_cluster.instances["s0_0_0"] pure_s3 = node.query( f""" SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) """ ) # print(pure_s3) s3_distributed = node.query( f""" SELECT * FROM ( SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) """ ) # print(s3_distributed) > s3_distributed_alt_syntax = node.query( f""" SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple' """ ) test_s3_cluster/test.py:241: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007fe48149eac3 E 16. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: SELECT * FROM E ( E SELECT * from s3( E 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E UNION ALL E SELECT * from s3( E 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E ) E ORDER BY (name, value, polygon) E SETTINGS object_storage_cluster = 'cluster_simple' E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * FROM ( SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:30 [ 684 ] DEBUG : Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:30 [ 684 ] DEBUG : Executing query SELECT * FROM ( SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:31 [ 684 ] DEBUG : Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) ______________________ test_unset_skip_unavailable_shards ______________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_unset_skip_unavailable_shards(started_cluster): # Although skip_unavailable_shards is not set, cluster table functions should always skip unavailable shards. node = started_cluster.instances["s0_0_0"] result = node.query( f""" SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') """ ) assert result == "10\n" > result = node.query( f""" SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_non_existent_port' """ ) test_s3_cluster/test.py:354: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007fe48149eac3 E 16. ? @ 0x00007fe481530850 E . (UNKNOWN_SETTING) E (query: SELECT count(*) from s3( E 'http://minio1:9001/root/data/clickhouse/part1.csv', E 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E SETTINGS object_storage_cluster = 'cluster_non_existent_port' E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_non_existent_port' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:31 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:30:31 [ 684 ] DEBUG : Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_non_existent_port' on s0_0_0 (cluster.py:3570, query) ______________________________ test_wrong_cluster ______________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_wrong_cluster(started_cluster): node = started_cluster.instances["s0_0_0"] error = node.query_and_get_error( f""" SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') """ ) assert "not found" in error error = node.query_and_get_error( f""" SELECT count(*) from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'non_existing_cluster' """ ) > assert "not found" in error E assert 'not found' in "Received exception from server (version 25.3.3):\nCode: 115. DB::Exception: Received from 172.16.7.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace:\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb\n1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c\n2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b\n3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8\n4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944\n5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3\n6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4\n7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303\n8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4\n9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9\n10. DB::TCPHandler::run() @ 0x0000000014cbbaf9\n11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7\n12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419\n13. Poco::PooledThread::run() @ 0x00000000183b975b\n14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d\n15. ? @ 0x00007fe48149eac3\n16. ? @ 0x00007fe481530850\n. (UNKNOWN_SETTING)\n(query: SELECT count(*) from s3(\n 'http://minio1:9001/root/data/{clickhouse,database}/*',\n 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')\n UNION ALL\n SELECT count(*) from s3(\n 'http://minio1:9001/root/data/{clickhouse,database}/*',\n 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')\n SETTINGS object_storage_cluster = 'non_existing_cluster'\n )\n" test_s3_cluster/test.py:293: AssertionError ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'non_existing_cluster' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:30:31 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3653, query_and_get_error) 2025-05-23 21:30:31 [ 684 ] DEBUG : Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'non_existing_cluster' on s0_0_0 (cluster.py:3653, query_and_get_error) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_1_0/docker-compose.yml stop --timeout 20] Stderr: Container roottests3cluster-gw0-resolver-1 Stopping Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopping Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopped Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopped Stderr: Container roottests3cluster-gw0-minio1-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopped Stderr: Container roottests3cluster-gw0-zoo1-1 Stopping Stderr: Container roottests3cluster-gw0-zoo2-1 Stopping Stderr: Container roottests3cluster-gw0-zoo3-1 Stopping Stderr: Container roottests3cluster-gw0-minio1-1 Stopped Stderr: Container roottests3cluster-gw0-zoo1-1 Stopped Stderr: Container roottests3cluster-gw0-zoo3-1 Stopped Stderr: Container roottests3cluster-gw0-zoo2-1 Stopped Stderr: Container roottests3cluster-gw0-resolver-1 Stopped Stderr: Container roottests3cluster-gw0-proxy1-1 Stopping Stderr: Container roottests3cluster-gw0-proxy2-1 Stopping Stderr: Container roottests3cluster-gw0-proxy2-1 Stopped Stderr: Container roottests3cluster-gw0-proxy1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_1_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_1_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_1_0/docker-compose.yml down --volumes] Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopping Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopping Stderr: Container roottests3cluster-gw0-resolver-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopped Stderr: Container roottests3cluster-gw0-s0_0_0-1 Removing Stderr: Container roottests3cluster-gw0-resolver-1 Stopped Stderr: Container roottests3cluster-gw0-resolver-1 Removing Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopped Stderr: Container roottests3cluster-gw0-s0_1_0-1 Removing Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopped Stderr: Container roottests3cluster-gw0-s0_0_1-1 Removing Stderr: Container roottests3cluster-gw0-s0_0_0-1 Removed Stderr: Container roottests3cluster-gw0-s0_0_1-1 Removed Stderr: Container roottests3cluster-gw0-minio1-1 Stopping Stderr: Container roottests3cluster-gw0-minio1-1 Stopped Stderr: Container roottests3cluster-gw0-minio1-1 Removing Stderr: Container roottests3cluster-gw0-resolver-1 Removed Stderr: Container roottests3cluster-gw0-s0_1_0-1 Removed Stderr: Container roottests3cluster-gw0-zoo1-1 Stopping Stderr: Container roottests3cluster-gw0-zoo2-1 Stopping Stderr: Container roottests3cluster-gw0-zoo3-1 Stopping Stderr: Container roottests3cluster-gw0-zoo1-1 Stopped Stderr: Container roottests3cluster-gw0-zoo1-1 Removing Stderr: Container roottests3cluster-gw0-zoo3-1 Stopped Stderr: Container roottests3cluster-gw0-zoo3-1 Removing Stderr: Container roottests3cluster-gw0-zoo2-1 Stopped Stderr: Container roottests3cluster-gw0-zoo2-1 Removing Stderr: Container roottests3cluster-gw0-zoo1-1 Removed Stderr: Container roottests3cluster-gw0-zoo2-1 Removed Stderr: Container roottests3cluster-gw0-zoo3-1 Removed Stderr: Container roottests3cluster-gw0-minio1-1 Removed Stderr: Container roottests3cluster-gw0-proxy1-1 Stopping Stderr: Container roottests3cluster-gw0-proxy2-1 Stopping Stderr: Container roottests3cluster-gw0-proxy1-1 Stopped Stderr: Container roottests3cluster-gw0-proxy1-1 Removing Stderr: Container roottests3cluster-gw0-proxy2-1 Stopped Stderr: Container roottests3cluster-gw0-proxy2-1 Removing Stderr: Container roottests3cluster-gw0-proxy2-1 Removed Stderr: Container roottests3cluster-gw0-proxy1-1 Removed Stderr: Volume roottests3cluster-gw0_data1-1 Removing Stderr: Network roottests3cluster-gw0_default Removing Stderr: Volume roottests3cluster-gw0_data1-1 Removed Stderr: Network roottests3cluster-gw0_default Removed Cleanup called Docker networks for project roottests3cluster-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3cluster-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottests3cluster-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottests3cluster-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottests3cluster-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:9 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 9 ---------------------------- Captured log teardown ----------------------------- 2025-05-23 21:30:32 [ 684 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_1_0/docker-compose.yml stop --timeout 20] (cluster.py:121, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_1_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_1_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw0/s0_1_0/docker-compose.yml down --volumes] (cluster.py:121, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Volume roottests3cluster-gw0_data1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Network roottests3cluster-gw0_default Removing (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Volume roottests3cluster-gw0_data1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stderr: Network roottests3cluster-gw0_default Removed (cluster.py:147, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Cleanup called (cluster.py:846, cleanup) 2025-05-23 21:30:53 [ 684 ] DEBUG : Docker networks for project roottests3cluster-gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:825, print_all_docker_pieces) 2025-05-23 21:30:53 [ 684 ] DEBUG : Docker containers for project roottests3cluster-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:833, print_all_docker_pieces) 2025-05-23 21:30:53 [ 684 ] DEBUG : Docker volumes for project roottests3cluster-gw0 are DRIVER VOLUME NAME (cluster.py:841, print_all_docker_pieces) 2025-05-23 21:30:53 [ 684 ] DEBUG : Command:[docker container list --all --filter name='^/roottests3cluster-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Unstopped containers: {} (cluster.py:860, cleanup) 2025-05-23 21:30:53 [ 684 ] DEBUG : No running containers for project: roottests3cluster-gw0 (cluster.py:874, cleanup) 2025-05-23 21:30:53 [ 684 ] DEBUG : Trying to prune unused networks... (cluster.py:880, cleanup) 2025-05-23 21:30:53 [ 684 ] DEBUG : Trying to prune unused images... (cluster.py:896, cleanup) 2025-05-23 21:30:53 [ 684 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Images pruned (cluster.py:899, cleanup) 2025-05-23 21:30:53 [ 684 ] DEBUG : Trying to prune unused volumes... (cluster.py:905, cleanup) 2025-05-23 21:30:53 [ 684 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stdout:9 (cluster.py:145, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2025-05-23 21:30:53 [ 684 ] DEBUG : Volumes pruned: 9 (cluster.py:910, cleanup) _____________________________ test_delete_cluster ______________________________ [gw9] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_delete_cluster(started_cluster): send_repeated_query("distributed") send_repeated_query("distributed2") > assert get_errors_count("test_cluster") > 0 E AssertionError: assert 0 > 0 E + where 0 = get_errors_count('test_cluster') test_reload_clusters_config/test.py:217: AssertionError ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node ------------------------------ Captured log call ------------------------------- 2025-05-23 21:32:40 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:32:52 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:33:04 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:33:16 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:33:29 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:33:41 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:33:54 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:34:06 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:34:19 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:34:31 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:34:43 [ 787 ] DEBUG : Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node (cluster.py:3570, query) ___________________________ test_update_one_cluster ____________________________ [gw9] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_update_one_cluster(started_cluster): send_repeated_query("distributed") send_repeated_query("distributed2") > assert get_errors_count("test_cluster") > 0 E AssertionError: assert 0 > 0 E + where 0 = get_errors_count('test_cluster') test_reload_clusters_config/test.py:202: AssertionError ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node ------------------------------ Captured log call ------------------------------- 2025-05-23 21:35:45 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:35:57 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:36:09 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:36:22 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:36:34 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:36:47 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:37:02 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:37:15 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:37:27 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:37:39 [ 787 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:37:51 [ 787 ] DEBUG : Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node (cluster.py:3570, query) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/.env --project-name roottestreloadclustersconfig-gw9 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] Stderr: Container roottestreloadclustersconfig-gw9-node-1 Stopping Stderr: Container roottestreloadclustersconfig-gw9-node-1 Stopped Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Stopping Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Stopping Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Stopping Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Stopped Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Stopped Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/.env --project-name roottestreloadclustersconfig-gw9 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] Stderr: Container roottestreloadclustersconfig-gw9-node-1 Stopping Stderr: Container roottestreloadclustersconfig-gw9-node-1 Stopped Stderr: Container roottestreloadclustersconfig-gw9-node-1 Removing Stderr: Container roottestreloadclustersconfig-gw9-node-1 Removed Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Stopping Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Stopping Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Stopping Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Stopped Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Removing Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Stopped Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Removing Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Stopped Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Removing Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Removed Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Removed Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Removed Stderr: Network roottestreloadclustersconfig-gw9_default Removing Stderr: Network roottestreloadclustersconfig-gw9_default Removed Cleanup called Docker networks for project roottestreloadclustersconfig-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreloadclustersconfig-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreloadclustersconfig-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreloadclustersconfig-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreloadclustersconfig-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 ---------------------------- Captured log teardown ----------------------------- 2025-05-23 21:37:51 [ 787 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/.env --project-name roottestreloadclustersconfig-gw9 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] (cluster.py:121, run_and_check) 2025-05-23 21:38:00 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-node-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:38:00 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-node-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:38:00 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:38:00 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:38:00 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:38:00 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:38:00 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:38:00 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:38:00 [ 787 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2025-05-23 21:38:00 [ 787 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/.env --project-name roottestreloadclustersconfig-gw9 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw9/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] (cluster.py:121, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-node-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-node-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-node-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-node-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo2-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo3-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw9-zoo1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Network roottestreloadclustersconfig-gw9_default Removing (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stderr: Network roottestreloadclustersconfig-gw9_default Removed (cluster.py:147, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Cleanup called (cluster.py:846, cleanup) 2025-05-23 21:38:01 [ 787 ] DEBUG : Docker networks for project roottestreloadclustersconfig-gw9 are NETWORK ID NAME DRIVER SCOPE (cluster.py:825, print_all_docker_pieces) 2025-05-23 21:38:01 [ 787 ] DEBUG : Docker containers for project roottestreloadclustersconfig-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:833, print_all_docker_pieces) 2025-05-23 21:38:01 [ 787 ] DEBUG : Docker volumes for project roottestreloadclustersconfig-gw9 are DRIVER VOLUME NAME (cluster.py:841, print_all_docker_pieces) 2025-05-23 21:38:01 [ 787 ] DEBUG : Command:[docker container list --all --filter name='^/roottestreloadclustersconfig-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Unstopped containers: {} (cluster.py:860, cleanup) 2025-05-23 21:38:01 [ 787 ] DEBUG : No running containers for project: roottestreloadclustersconfig-gw9 (cluster.py:874, cleanup) 2025-05-23 21:38:01 [ 787 ] DEBUG : Trying to prune unused networks... (cluster.py:880, cleanup) 2025-05-23 21:38:01 [ 787 ] DEBUG : Trying to prune unused images... (cluster.py:896, cleanup) 2025-05-23 21:38:01 [ 787 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Images pruned (cluster.py:899, cleanup) 2025-05-23 21:38:01 [ 787 ] DEBUG : Trying to prune unused volumes... (cluster.py:905, cleanup) 2025-05-23 21:38:01 [ 787 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2025-05-23 21:38:01 [ 787 ] DEBUG : Volumes pruned: 1 (cluster.py:910, cleanup) ----------------- generated report log file: parallel2_0.jsonl ----------------- ============================== slowest durations =============================== 139.06s call test_reload_clusters_config/test.py::test_add_cluster 138.00s setup test_postgresql_protocol/test.py::test_java_client 126.05s call test_reload_clusters_config/test.py::test_update_one_cluster 123.45s call test_reload_clusters_config/test.py::test_delete_cluster 61.46s call test_reload_clusters_config/test.py::test_simple_reload 28.73s setup test_storage_mongodb_legacy/test.py::test_secure_connection[True] 28.17s setup test_storage_mongodb_legacy/test.py::test_arrays[False] 26.70s setup test_reload_clusters_config/test.py::test_add_cluster 24.01s setup test_s3_cluster/test.py::test_ambiguous_join 21.97s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format 21.72s teardown test_s3_cluster/test.py::test_wrong_cluster 20.89s teardown test_postgresql_protocol/test.py::test_python_client 20.78s setup test_replication_credentials/test.py::test_credentials_and_no_credentials 20.69s setup test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] 19.41s setup test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_auxiliary_zookeeper 19.35s teardown test_quorum_inserts/test.py::test_simple_add_replica 18.64s call test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection 18.35s setup test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility 18.18s setup test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum 17.21s call test_quorum_inserts/test.py::test_insert_quorum_with_ttl 16.66s setup test_replicated_merge_tree_thread_schedule_timeouts/test.py::test_query_exception_on_thread_pool_full 16.44s teardown test_permissions_drop_replica/test.py::test_drop_permissions 15.78s setup test_shard_level_const_function/test.py::test_remote 15.57s setup test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers 15.22s setup test_permissions_drop_replica/test.py::test_drop_permissions 14.86s teardown test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility 13.65s setup test_session_log/test.py::test_grpc_session 13.26s call test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] 12.98s setup test_ssl_cert_authentication/test.py::test_create_user 11.69s setup test_rocksdb_options/test.py::test_invalid_block_based_table_options 11.56s teardown test_replication_credentials/test.py::test_same_credentials 11.05s call test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers 10.66s teardown test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys 10.13s teardown test_reload_clusters_config/test.py::test_update_one_cluster 9.91s setup test_sharding_key_from_default_column/test.py::test_default_column 9.71s setup test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys 9.40s call test_rocksdb_options/test.py::test_table_invalid_options 9.37s call test_rocksdb_options/test.py::test_invalid_options 9.27s call test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 9.25s call test_rocksdb_options/test.py::test_table_invalid_column_family_options 8.50s call test_rocksdb_options/test.py::test_invalid_block_based_table_options 8.17s call test_rocksdb_options/test.py::test_invalid_column_family_options 8.16s teardown test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized_different_shards 8.13s call test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility 6.61s setup test_part_log_table/test.py::test_config_without_part_log 6.44s teardown test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers 6.38s call test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_auxiliary_zookeeper 6.28s teardown test_shard_level_const_function/test.py::test_remote 6.26s call test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_drop_replicated_merge_tree_with_auxiliary_zookeeper 6.23s setup test_settings_from_server/test.py::test_settings_from_server 5.79s call test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum 5.61s call test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_default_zookeeper 5.56s teardown test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_path_ambiguity 5.55s call test_permissions_drop_replica/test.py::test_drop_permissions 4.40s teardown test_replicated_merge_tree_thread_schedule_timeouts/test.py::test_query_exception_on_thread_pool_full 4.17s teardown test_session_log/test.py::test_postgres_session 4.12s teardown test_storage_mongodb_legacy/test.py::test_secure_connection[True] 4.01s teardown test_settings_randomization/test.py::test_settings_randomization 3.82s teardown test_part_log_table/test.py::test_part_log_contains_partition 3.63s teardown test_settings_from_server/test.py::test_settings_from_server 3.52s call test_replicated_merge_tree_thread_schedule_timeouts/test.py::test_query_exception_on_thread_pool_full 2.95s call test_sharding_key_from_default_column/test.py::test_materialized_column_allow_insert_materialized 2.81s call test_sharding_key_from_default_column/test.py::test_default_column 2.75s call test_replication_credentials/test.py::test_different_credentials 2.75s call test_replication_credentials/test.py::test_credentials_and_no_credentials 2.65s call test_replication_credentials/test.py::test_same_credentials 2.56s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] 2.54s call test_replication_credentials/test.py::test_no_credentials 2.35s call test_storage_mongodb_legacy/test.py::test_arrays[False] 2.26s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] 2.21s teardown test_ssl_cert_authentication/test.py::test_x509_san_wildcard_support 2.12s setup test_settings_randomization/test.py::test_settings_randomization 2.08s call test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized_different_shards 2.07s call test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys 1.93s call test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized 1.90s call test_s3_cluster/test.py::test_hive_partitioning[1] 1.89s call test_s3_cluster/test.py::test_hive_partitioning[0] 1.70s call test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys 1.59s call test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] 1.48s call test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] 1.38s call test_settings_from_server/test.py::test_settings_from_server 1.30s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] 1.28s call test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 1.26s call test_session_log/test.py::test_parallel_sessions 1.25s call test_session_log/test.py::test_postgres_session 1.23s call test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format 1.21s call test_quorum_inserts/test.py::test_simple_add_replica 1.19s teardown test_rocksdb_options/test.py::test_valid_options 1.10s call test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] 1.04s call test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 1.00s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] 1.00s call test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] 0.97s call test_storage_mongodb_legacy/test.py::test_secure_connection[True] 0.90s call test_ssl_cert_authentication/test.py::test_create_user 0.88s call test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_path_ambiguity 0.85s call test_postgresql_protocol/test.py::test_psql_client 0.85s call test_s3_cluster/test.py::test_cluster_default_expression 0.75s call test_ssl_cert_authentication/test.py::test_x509_san_wildcard_support 0.75s call test_part_log_table/test.py::test_config_without_part_log 0.75s call test_s3_cluster/test.py::test_distributed_s3_table_engine 0.73s call test_session_log/test.py::test_grpc_session 0.70s call test_s3_cluster/test.py::test_cluster_format_detection 0.69s call test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 0.60s call test_storage_mongodb_legacy/test.py::test_complex_data_type[False] 0.58s call test_rocksdb_options/test.py::test_valid_options 0.57s call test_session_log/test.py::test_mysql_session 0.55s call test_s3_cluster/test.py::test_select_all 0.55s call test_ssl_cert_authentication/test.py::test_x509_san_support 0.51s call test_storage_mongodb_legacy/test.py::test_incorrect_data_type[False] 0.50s call test_storage_mongodb_legacy/test.py::test_auth_source[False] 0.48s call test_s3_cluster/test.py::test_unset_skip_unavailable_shards 0.47s call test_postgresql_protocol/test.py::test_java_client 0.46s call test_part_log_table/test.py::test_part_log_contains_partition 0.42s call test_s3_cluster/test.py::test_cluster_with_header 0.41s call test_s3_cluster/test.py::test_cluster_with_named_collection 0.39s setup test_replication_credentials/test.py::test_different_credentials 0.38s call test_s3_cluster/test.py::test_wrong_cluster 0.38s call test_storage_mongodb_legacy/test.py::test_missing_columns[False] 0.36s setup test_replication_credentials/test.py::test_same_credentials 0.35s call test_s3_cluster/test.py::test_union_all 0.33s teardown test_sharding_key_from_default_column/test.py::test_default_column 0.32s call test_storage_mongodb_legacy/test.py::test_predefined_connection_configuration[False] 0.30s call test_s3_cluster/test.py::test_count_macro 0.30s call test_ssl_cert_authentication/test.py::test_native 0.29s setup test_replication_credentials/test.py::test_no_credentials 0.29s teardown test_sharding_key_from_default_column/test.py::test_materialized_column_allow_insert_materialized 0.23s call test_s3_cluster/test.py::test_count 0.23s teardown test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized 0.23s call test_s3_cluster/test.py::test_remote_no_hedged 0.23s call test_s3_cluster/test.py::test_skip_unavailable_shards 0.22s call test_storage_mongodb_legacy/test.py::test_no_credentials[False] 0.22s call test_s3_cluster/test.py::test_ambiguous_join 0.20s call test_settings_randomization/test.py::test_settings_randomization 0.18s call test_s3_cluster/test.py::test_cluster_hosts_limit 0.18s call test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_not_exists_auxiliary_zookeeper 0.18s call test_ssl_cert_authentication/test.py::test_native_fallback_to_password 0.13s call test_ssl_cert_authentication/test.py::test_https_non_ssl_auth 0.13s call test_ssl_cert_authentication/test.py::test_native_wrong_cert 0.12s call test_shard_level_const_function/test.py::test_remote 0.07s call test_rocksdb_options/test.py::test_table_valid_options 0.07s call test_s3_cluster/test.py::test_remote_hedged 0.07s call test_rocksdb_options/test.py::test_valid_column_family_options 0.07s call test_rocksdb_options/test.py::test_table_valid_column_family_options 0.06s call test_ssl_cert_authentication/test.py::test_https 0.04s call test_ssl_cert_authentication/test.py::test_https_wrong_cert 0.02s call test_postgresql_protocol/test.py::test_prepared_statement 0.01s call test_postgresql_protocol/test.py::test_python_client 0.00s teardown test_storage_mongodb_legacy/test.py::test_auth_source[False] 0.00s setup test_sharding_key_from_default_column/test.py::test_materialized_column_allow_insert_materialized 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] 0.00s setup test_reload_clusters_config/test.py::test_delete_cluster 0.00s teardown test_reload_clusters_config/test.py::test_add_cluster 0.00s setup test_storage_mongodb_legacy/test.py::test_complex_data_type[False] 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format 0.00s teardown test_s3_cluster/test.py::test_ambiguous_join 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[1] 0.00s teardown test_replication_credentials/test.py::test_credentials_and_no_credentials 0.00s teardown test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 0.00s teardown test_s3_cluster/test.py::test_remote_no_hedged 0.00s setup test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_drop_replicated_merge_tree_with_auxiliary_zookeeper 0.00s setup test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] 0.00s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] 0.00s teardown test_replication_credentials/test.py::test_no_credentials 0.00s setup test_storage_mongodb_legacy/test.py::test_incorrect_data_type[False] 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] 0.00s setup test_s3_cluster/test.py::test_select_all 0.00s setup test_rocksdb_options/test.py::test_table_valid_options 0.00s setup test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized_different_shards 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[0] 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] 0.00s setup test_storage_mongodb_legacy/test.py::test_no_credentials[False] 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] 0.00s setup test_rocksdb_options/test.py::test_valid_options 0.00s setup test_s3_cluster/test.py::test_count_macro 0.00s setup test_part_log_table/test.py::test_part_log_contains_partition 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] 0.00s setup test_storage_mongodb_legacy/test.py::test_missing_columns[False] 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] 0.00s setup test_storage_mongodb_legacy/test.py::test_predefined_connection_configuration[False] 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_ttl 0.00s setup test_session_log/test.py::test_parallel_sessions 0.00s setup test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_default_zookeeper 0.00s teardown test_s3_cluster/test.py::test_union_all 0.00s teardown test_rocksdb_options/test.py::test_invalid_options 0.00s setup test_postgresql_protocol/test.py::test_psql_client 0.00s setup test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys 0.00s teardown test_s3_cluster/test.py::test_distributed_s3_table_engine 0.00s teardown test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] 0.00s teardown test_s3_cluster/test.py::test_count 0.00s teardown test_postgresql_protocol/test.py::test_java_client 0.00s setup test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_not_exists_auxiliary_zookeeper 0.00s teardown test_storage_mongodb_legacy/test.py::test_complex_data_type[False] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[0] 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] 0.00s teardown test_part_log_table/test.py::test_config_without_part_log 0.00s teardown test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_not_exists_auxiliary_zookeeper 0.00s teardown test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s teardown test_storage_mongodb_legacy/test.py::test_arrays[False] 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] 0.00s teardown test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_auxiliary_zookeeper 0.00s setup test_ssl_cert_authentication/test.py::test_https_non_ssl_auth 0.00s setup test_postgresql_protocol/test.py::test_prepared_statement 0.00s setup test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s setup test_s3_cluster/test.py::test_union_all 0.00s setup test_s3_cluster/test.py::test_cluster_with_header 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] 0.00s teardown test_session_log/test.py::test_parallel_sessions 0.00s teardown test_rocksdb_options/test.py::test_valid_column_family_options 0.00s setup test_s3_cluster/test.py::test_remote_hedged 0.00s teardown test_s3_cluster/test.py::test_unset_skip_unavailable_shards 0.00s setup test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[1] 0.00s setup test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 0.00s teardown test_postgresql_protocol/test.py::test_prepared_statement 0.00s setup test_session_log/test.py::test_postgres_session 0.00s teardown test_ssl_cert_authentication/test.py::test_native 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection 0.00s setup test_ssl_cert_authentication/test.py::test_native_fallback_to_password 0.00s teardown test_rocksdb_options/test.py::test_invalid_block_based_table_options 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 0.00s setup test_s3_cluster/test.py::test_unset_skip_unavailable_shards 0.00s setup test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_path_ambiguity 0.00s teardown test_reload_clusters_config/test.py::test_delete_cluster 0.00s setup test_reload_clusters_config/test.py::test_simple_reload 0.00s teardown test_s3_cluster/test.py::test_cluster_format_detection 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s teardown test_s3_cluster/test.py::test_select_all 0.00s teardown test_s3_cluster/test.py::test_remote_hedged 0.00s teardown test_s3_cluster/test.py::test_cluster_with_header 0.00s setup test_rocksdb_options/test.py::test_table_invalid_options 0.00s setup test_s3_cluster/test.py::test_count 0.00s teardown test_s3_cluster/test.py::test_skip_unavailable_shards 0.00s setup test_ssl_cert_authentication/test.py::test_x509_san_support 0.00s setup test_postgresql_protocol/test.py::test_python_client 0.00s teardown test_session_log/test.py::test_mysql_session 0.00s teardown test_ssl_cert_authentication/test.py::test_create_user 0.00s setup test_ssl_cert_authentication/test.py::test_x509_san_wildcard_support 0.00s setup test_rocksdb_options/test.py::test_valid_column_family_options 0.00s setup test_s3_cluster/test.py::test_distributed_s3_table_engine 0.00s setup test_s3_cluster/test.py::test_cluster_default_expression 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] 0.00s teardown test_rocksdb_options/test.py::test_table_valid_options 0.00s teardown test_storage_mongodb_legacy/test.py::test_incorrect_data_type[False] 0.00s teardown test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_drop_replicated_merge_tree_with_auxiliary_zookeeper 0.00s teardown test_s3_cluster/test.py::test_count_macro 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] 0.00s teardown test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s teardown test_postgresql_protocol/test.py::test_psql_client 0.00s setup test_s3_cluster/test.py::test_cluster_hosts_limit 0.00s setup test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s setup test_s3_cluster/test.py::test_skip_unavailable_shards 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] 0.00s setup test_s3_cluster/test.py::test_cluster_with_named_collection 0.00s setup test_s3_cluster/test.py::test_wrong_cluster 0.00s setup test_storage_mongodb_legacy/test.py::test_auth_source[False] 0.00s setup test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection 0.00s setup test_ssl_cert_authentication/test.py::test_https 0.00s setup test_reload_clusters_config/test.py::test_update_one_cluster 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_options 0.00s teardown test_s3_cluster/test.py::test_cluster_with_named_collection 0.00s setup test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s setup test_session_log/test.py::test_mysql_session 0.00s teardown test_s3_cluster/test.py::test_cluster_hosts_limit 0.00s setup test_ssl_cert_authentication/test.py::test_native_wrong_cert 0.00s setup test_rocksdb_options/test.py::test_invalid_options 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] 0.00s teardown test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_default_zookeeper 0.00s setup test_ssl_cert_authentication/test.py::test_https_wrong_cert 0.00s setup test_quorum_inserts/test.py::test_simple_add_replica 0.00s teardown test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 0.00s teardown test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 0.00s teardown test_session_log/test.py::test_grpc_session 0.00s setup test_ssl_cert_authentication/test.py::test_native 0.00s teardown test_storage_mongodb_legacy/test.py::test_no_credentials[False] 0.00s teardown test_ssl_cert_authentication/test.py::test_x509_san_support 0.00s setup test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 0.00s setup test_s3_cluster/test.py::test_remote_no_hedged 0.00s teardown test_ssl_cert_authentication/test.py::test_https 0.00s teardown test_s3_cluster/test.py::test_cluster_default_expression 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 0.00s teardown test_quorum_inserts/test.py::test_insert_quorum_with_ttl 0.00s teardown test_storage_mongodb_legacy/test.py::test_missing_columns[False] 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] 0.00s setup test_s3_cluster/test.py::test_cluster_format_detection 0.00s teardown test_ssl_cert_authentication/test.py::test_native_wrong_cert 0.00s teardown test_ssl_cert_authentication/test.py::test_https_wrong_cert 0.00s teardown test_storage_mongodb_legacy/test.py::test_predefined_connection_configuration[False] 0.00s teardown test_reload_clusters_config/test.py::test_simple_reload 0.00s call test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] 0.00s teardown test_ssl_cert_authentication/test.py::test_https_non_ssl_auth 0.00s teardown test_replication_credentials/test.py::test_different_credentials 0.00s teardown test_ssl_cert_authentication/test.py::test_native_fallback_to_password 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] =========================== short test summary info ============================ FAILED test_s3_cluster/test.py::test_cluster_default_expression - helpers.cli... FAILED test_s3_cluster/test.py::test_cluster_format_detection - helpers.clien... FAILED test_s3_cluster/test.py::test_cluster_hosts_limit - helpers.client.Que... FAILED test_s3_cluster/test.py::test_cluster_with_header - helpers.client.Que... FAILED test_s3_cluster/test.py::test_cluster_with_named_collection - helpers.... FAILED test_s3_cluster/test.py::test_count - NameError: name 'clickhouse' is ... FAILED test_s3_cluster/test.py::test_count_macro - helpers.client.QueryRuntim... FAILED test_s3_cluster/test.py::test_distributed_s3_table_engine - helpers.cl... FAILED test_s3_cluster/test.py::test_hive_partitioning[0] - assert 2984 == 1492 FAILED test_s3_cluster/test.py::test_hive_partitioning[1] - assert 2984 == 1492 FAILED test_s3_cluster/test.py::test_remote_hedged - helpers.client.QueryRunt... FAILED test_s3_cluster/test.py::test_remote_no_hedged - helpers.client.QueryR... FAILED test_s3_cluster/test.py::test_select_all - helpers.client.QueryRuntime... FAILED test_s3_cluster/test.py::test_skip_unavailable_shards - helpers.client... FAILED test_s3_cluster/test.py::test_union_all - helpers.client.QueryRuntimeE... FAILED test_s3_cluster/test.py::test_unset_skip_unavailable_shards - helpers.... FAILED test_s3_cluster/test.py::test_wrong_cluster - assert 'not found' in "R... FAILED test_reload_clusters_config/test.py::test_delete_cluster - AssertionEr... FAILED test_reload_clusters_config/test.py::test_update_one_cluster - Asserti... PASSED test_ssl_cert_authentication/test.py::test_create_user PASSED test_ssl_cert_authentication/test.py::test_https PASSED test_ssl_cert_authentication/test.py::test_https_non_ssl_auth PASSED test_ssl_cert_authentication/test.py::test_https_wrong_cert PASSED test_ssl_cert_authentication/test.py::test_native PASSED test_ssl_cert_authentication/test.py::test_native_fallback_to_password PASSED test_ssl_cert_authentication/test.py::test_native_wrong_cert PASSED test_ssl_cert_authentication/test.py::test_x509_san_support PASSED test_ssl_cert_authentication/test.py::test_x509_san_wildcard_support PASSED test_rocksdb_options/test.py::test_invalid_block_based_table_options PASSED test_replication_credentials/test.py::test_credentials_and_no_credentials PASSED test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum PASSED test_s3_cluster/test.py::test_ambiguous_join PASSED test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False] PASSED test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_auxiliary_zookeeper PASSED test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True] PASSED test_replication_credentials/test.py::test_different_credentials PASSED test_rocksdb_options/test.py::test_invalid_column_family_options PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated PASSED test_replication_credentials/test.py::test_no_credentials PASSED test_storage_mongodb_legacy/test.py::test_arrays[False] PASSED test_storage_mongodb_legacy/test.py::test_auth_source[False] PASSED test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_default_zookeeper PASSED test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_not_exists_auxiliary_zookeeper PASSED test_storage_mongodb_legacy/test.py::test_complex_data_type[False] PASSED test_storage_mongodb_legacy/test.py::test_incorrect_data_type[False] PASSED test_replication_credentials/test.py::test_same_credentials PASSED test_storage_mongodb_legacy/test.py::test_missing_columns[False] PASSED test_session_log/test.py::test_grpc_session PASSED test_storage_mongodb_legacy/test.py::test_no_credentials[False] PASSED test_storage_mongodb_legacy/test.py::test_predefined_connection_configuration[False] PASSED test_session_log/test.py::test_mysql_session PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] PASSED test_session_log/test.py::test_parallel_sessions PASSED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] PASSED test_session_log/test.py::test_postgres_session PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] PASSED test_rocksdb_options/test.py::test_invalid_options PASSED test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_drop_replicated_merge_tree_with_auxiliary_zookeeper PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] PASSED test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_path_ambiguity PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format PASSED test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection PASSED test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False] PASSED test_rocksdb_options/test.py::test_table_invalid_block_based_table_options PASSED test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True] PASSED test_part_log_table/test.py::test_config_without_part_log PASSED test_part_log_table/test.py::test_part_log_contains_partition PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options PASSED test_sharding_key_from_default_column/test.py::test_default_column PASSED test_sharding_key_from_default_column/test.py::test_materialized_column_allow_insert_materialized PASSED test_permissions_drop_replica/test.py::test_drop_permissions PASSED test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized PASSED test_storage_mongodb_legacy/test.py::test_secure_connection[True] PASSED test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized_different_shards PASSED test_quorum_inserts/test.py::test_insert_quorum_with_ttl PASSED test_rocksdb_options/test.py::test_table_invalid_options PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options PASSED test_rocksdb_options/test.py::test_table_valid_options PASSED test_rocksdb_options/test.py::test_valid_column_family_options PASSED test_quorum_inserts/test.py::test_simple_add_replica PASSED test_rocksdb_options/test.py::test_valid_options PASSED test_settings_from_server/test.py::test_settings_from_server PASSED test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys PASSED test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys PASSED test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility PASSED test_shard_level_const_function/test.py::test_remote PASSED test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers PASSED test_settings_randomization/test.py::test_settings_randomization PASSED test_replicated_merge_tree_thread_schedule_timeouts/test.py::test_query_exception_on_thread_pool_full PASSED test_postgresql_protocol/test.py::test_java_client PASSED test_postgresql_protocol/test.py::test_prepared_statement PASSED test_postgresql_protocol/test.py::test_psql_client PASSED test_postgresql_protocol/test.py::test_python_client PASSED test_reload_clusters_config/test.py::test_add_cluster PASSED test_reload_clusters_config/test.py::test_simple_reload ================== 19 failed, 81 passed in 489.34s (0:08:09) =================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_lx8764 --privileged --dns-search='.' --memory=30709035008 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel2_0.jsonl --report-log-exclude-logs-on-passed-tests test_part_log_table/test.py::test_config_without_part_log test_part_log_table/test.py::test_part_log_contains_partition test_permissions_drop_replica/test.py::test_drop_permissions test_postgresql_protocol/test.py::test_java_client test_postgresql_protocol/test.py::test_prepared_statement test_postgresql_protocol/test.py::test_psql_client test_postgresql_protocol/test.py::test_python_client test_quorum_inserts/test.py::test_drop_replica_and_achieve_quorum 'test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[False]' 'test_quorum_inserts/test.py::test_insert_quorum_with_drop_partition[True]' test_quorum_inserts/test.py::test_insert_quorum_with_keeper_loss_connection 'test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[False]' 'test_quorum_inserts/test.py::test_insert_quorum_with_move_partition[True]' test_quorum_inserts/test.py::test_insert_quorum_with_ttl test_quorum_inserts/test.py::test_simple_add_replica test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers test_reload_clusters_config/test.py::test_add_cluster test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_simple_reload test_reload_clusters_config/test.py::test_update_one_cluster 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys test_replicated_merge_tree_thread_schedule_timeouts/test.py::test_query_exception_on_thread_pool_full test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_auxiliary_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_default_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_create_replicated_merge_tree_with_not_exists_auxiliary_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_drop_replicated_merge_tree_with_auxiliary_zookeeper test_replicated_merge_tree_with_auxiliary_zookeepers/test.py::test_path_ambiguity test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_hosts_limit test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_distributed_s3_table_engine 'test_s3_cluster/test.py::test_hive_partitioning[0]' 'test_s3_cluster/test.py::test_hive_partitioning[1]' test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster test_session_log/test.py::test_grpc_session test_session_log/test.py::test_mysql_session test_session_log/test.py::test_parallel_sessions test_session_log/test.py::test_postgres_session test_settings_from_server/test.py::test_settings_from_server test_settings_randomization/test.py::test_settings_randomization test_shard_level_const_function/test.py::test_remote test_sharding_key_from_default_column/test.py::test_default_column test_sharding_key_from_default_column/test.py::test_materialized_column_allow_insert_materialized test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized test_sharding_key_from_default_column/test.py::test_materialized_column_disallow_insert_materialized_different_shards test_ssl_cert_authentication/test.py::test_create_user test_ssl_cert_authentication/test.py::test_https test_ssl_cert_authentication/test.py::test_https_non_ssl_auth test_ssl_cert_authentication/test.py::test_https_wrong_cert test_ssl_cert_authentication/test.py::test_native test_ssl_cert_authentication/test.py::test_native_fallback_to_password test_ssl_cert_authentication/test.py::test_native_wrong_cert test_ssl_cert_authentication/test.py::test_x509_san_support test_ssl_cert_authentication/test.py::test_x509_san_wildcard_support 'test_storage_mongodb_legacy/test.py::test_arrays[False]' 'test_storage_mongodb_legacy/test.py::test_auth_source[False]' 'test_storage_mongodb_legacy/test.py::test_complex_data_type[False]' 'test_storage_mongodb_legacy/test.py::test_incorrect_data_type[False]' 'test_storage_mongodb_legacy/test.py::test_missing_columns[False]' 'test_storage_mongodb_legacy/test.py::test_no_credentials[False]' 'test_storage_mongodb_legacy/test.py::test_predefined_connection_configuration[False]' 'test_storage_mongodb_legacy/test.py::test_secure_connection[True]' -vvv " altinityinfra/integration-tests-runner:ad96270260ff ' returned non-zero exit status 1.